{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLM APIs and Ollama\n",
    "\n",
    "_IMPORTANT: If you're not as familiar with APIs in general, and with Environment Variables on your PC or Mac, please review the APIs section in Guide 4 Technical Foundations before proceeding with this guide!_\n",
    "\n",
    "## Introduction to LLM APIs\n",
    "\n",
    "Throughout the course, we use APIs for connecting with the strongest LLMs on the planet.\n",
    "\n",
    "OpenAI has a python client library which simplifies the way we call OpenAI on the cloud:\n",
    "\n",
    "```python\n",
    "# Specify which model to use, and the prompts\n",
    "MODEL = \"gpt-4o-mini\"\n",
    "messages = [{\"role\": \"user\", \"content\": \"what is 2+2?\"}]\n",
    "\n",
    "# Create an OpenAI python client for making web calls to OpenAI\n",
    "openai = OpenAI()\n",
    "\n",
    "# Make the call\n",
    "response = openai.chat.completions.create(model=MODEL, messages=messages)\n",
    "print(response.choices[0].message.content)\n",
    "```\n",
    "\n",
    "Several other LLMs, including Google Gemini and DeepSeek, have API endpoints that are compatible with OpenAI. In fact, it's almost everyone aside from Anthropic!\n",
    "\n",
    "OpenAI has made their python client library available for others to use by switching the URL from their URL to somewhere else:\n",
    "\n",
    "`not_actually_openai = OpenAI(base_url=\"https://somewhere.completely.different/\", api_key=\"another_providers_key\")`\n",
    "\n",
    "It's important to realize that this OpenAI code is just a utility for making https calls to endpoints. There's no LLM code here - just a wrapper around a network call.\n",
    "\n",
    "Many API providers offer an OpenAI compatible endpoint that you can use as your base_url. Here are some popular ones:\n",
    "\n",
    "```python\n",
    "DEEPSEEK_BASE_URL = \"https://api.deepseek.com/v1\"\n",
    "GEMINI_BASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
    "GROK_BASE_URL = \"https://api.x.ai/v1\"\n",
    "GROQ_BASE_URL = \"https://api.groq.com/openai/v1\"\n",
    "OPENROUTER_BASE_URL = \"https://openrouter.ai/api/v1\"\n",
    "OLLAMA_BASE_URL = \"http://localhost:11434/v1\"\n",
    "```\n",
    "\n",
    "### Using different API providers with Agent Frameworks\n",
    "\n",
    "The Agent Frameworks make it easy to switch between these providers. You can switch LLMs and pick different ones at any point in the course. You may need to look up in the framework docs how best to switch to a different API, or ask me. For OpenAI Agents SDK, see a section later in this notebook. For CrewAI, we cover it on the course, but it's easy: just use the full path to the model that LiteLLM expects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Costs of APIs\n",
    "\n",
    "The cost of each API call is very low indeed - most calls to models we use on this course are fractions of cents.\n",
    "\n",
    "But it's extremely important to note:\n",
    "\n",
    "1. A complex Agentic project could involve many LLM calls - perhaps 20-30 - and so it can add up. It's important to set limits and monitor usage.\n",
    "\n",
    "2. With Agentic AI, there is a risk of Agents getting into a loop or carrying out more processing than intended. You should monitor your API usage, and never put more budget than you are comfortable with. Some APIs have an \"auto-refill\" setting that can charge automatically to your card - I strongly recommend you keep this off.\n",
    "\n",
    "3. You should only spend what you are comfortable with. There is a free alternative in Ollama that you can use as a replacement if you wish. DeepSeek, Gemini 2.5 Flash and gpt-4.1-nano are significantly cheaper.\n",
    "\n",
    "Keep in mind that these LLM calls typically involve trillions of floating point calculations - someone has to pay the electricity bills!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ollama: Free alternative to Paid APIs (but please see Warning about llama version)\n",
    "\n",
    "Ollama is a product that runs locally on your machine. It can run open-source models, and it provides an API endpoint on your computer that is compatible with OpenAI.\n",
    "\n",
    "First, download Ollama by visiting:\n",
    "https://ollama.com\n",
    "\n",
    "Then from your Terminal in Cursor (View menu >> Terminal), run this command to download a model:\n",
    "\n",
    "`ollama pull llama3.2`\n",
    "\n",
    "WARNING: Be careful not to use llama3.3 or llama4 - these are much larger models that are not suitable for home computers.\n",
    "\n",
    "And now, any time that we have code like:  \n",
    "`openai = OpenAI()`  \n",
    "You can use this as a direct replacement:  \n",
    "`openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')`  \n",
    "And also replace model names like **gpt-4o-mini** with **llama3.2**.  \n",
    "\n",
    "You don't need to put anything in your .env file for this; with Ollama, everything is running on your computer. You're not calling out to a third party on the cloud, nobody has your credit card details, so there's no need for a secret key! The code `api_key='ollama'` above is only required because the OpenAI client library expects an api_key to be passed in, but the value is ignored by Ollama.\n",
    "\n",
    "Below is a full example:\n",
    "\n",
    "```python\n",
    "# You need to do this one time on your computer\n",
    "!ollama pull llama3.2\n",
    "\n",
    "from openai import OpenAI\n",
    "MODEL = \"llama3.2\"\n",
    "openai = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\n",
    "\n",
    "response = openai.chat.completions.create(\n",
    " model=MODEL,\n",
    " messages=[{\"role\": \"user\", \"content\": \"What is 2 + 2?\"}]\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)\n",
    "```\n",
    "\n",
    "You will need to make similar changes to use Ollama within any of the Agent Frameworks - you should be able to google for an exact example, or ask me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenRouter: Convenient gateway platform for OpenAI and others\n",
    "\n",
    "OpenRouter is a third party service that allows you to connect to a wide range of LLMs, including OpenAI.\n",
    "\n",
    "It's known for having a simpler billing process that may be easier for some countries outside the US.\n",
    "\n",
    "First, check out their website:  \n",
    "https://openrouter.ai/\n",
    "\n",
    "Then, take a peak at their quickstart:  \n",
    "https://openrouter.ai/docs/quickstart\n",
    "\n",
    "And add your key to your .env file:  \n",
    "`OPENROUTER_API_KEY=sk-or....`\n",
    "\n",
    "And now, any time you have code like this:  \n",
    "```python\n",
    "MODEL = \"gpt-4o-mini\"\n",
    "openai = OpenAI()\n",
    "```\n",
    "\n",
    "You can replace it with code like this:\n",
    "\n",
    "```python\n",
    "MODEL = \"openai/gpt-4o-mini\"\n",
    "openrouter_api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
    "openai = OpenAI(base_url=\"https://openrouter.ai/api/v1\", api_key=openrouter_api_key)\n",
    "\n",
    "response = openai.chat.completions.create(\n",
    " model=MODEL,\n",
    " messages=[{\"role\": \"user\", \"content\": \"What is 2 + 2?\"}]\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)\n",
    "```\n",
    "\n",
    "You will need to make similar changes to use OpenRouter within any of the Agent Frameworks - you should be able to google for an exact example, or ask me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenAI Agents SDK - specific instructions\n",
    "\n",
    "With OpenAI Agents SDK (weeks 2 and 6), it's particularly easy to use any model provided by OpenAI themselves. Simply pass in the model name:\n",
    "\n",
    "`agent = Agent(name=\"Jokester\", instructions=\"You are a joke teller\", model=\"gpt-4o-mini\")`\n",
    "\n",
    "You can also substitute in any other provider with an OpenAI compatible API. You do it in 3 steps like this:\n",
    "\n",
    "```python\n",
    "DEEPSEEK_BASE_URL = \"https://api.deepseek.com/v1\"\n",
    "deepseek_client = AsyncOpenAI(base_url=DEEPSEEK_BASE_URL, api_key=deepseek_api_key)\n",
    "deepseek_model = OpenAIChatCompletionsModel(model=\"deepseek-chat\", openai_client=deepseek_client)\n",
    "```\n",
    "\n",
    "And then you simply provide this model when you create an Agent.\n",
    "\n",
    "`agent = Agent(name=\"Jokester\", instructions=\"You are a joke teller\", model=deepseek_model)`\n",
    "\n",
    "And you can use a similar approach for any other OpenAI compatible API, with the same 3 steps:\n",
    "\n",
    "```python\n",
    "# Specify the base URL endpoints where the provider offers an OpenAI compatible API\n",
    "GEMINI_BASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
    "GROK_BASE_URL = \"https://api.x.ai/v1\"\n",
    "GROQ_BASE_URL = \"https://api.groq.com/openai/v1\"\n",
    "OPENROUTER_BASE_URL = \"https://openrouter.ai/api/v1\"\n",
    "OLLAMA_BASE_URL = \"http://localhost:11434/v1\"\n",
    "\n",
    "# Create an AsyncOpenAI object for that endpoint\n",
    "gemini_client = AsyncOpenAI(base_url=GEMINI_BASE_URL, api_key=google_api_key)\n",
    "grok_client = AsyncOpenAI(base_url=GROK_BASE_URL, api_key=grok_api_key)\n",
    "groq_client = AsyncOpenAI(base_url=GROQ_BASE_URL, api_key=groq_api_key)\n",
    "openrouter_client = AsyncOpenAI(base_url=OPENROUTER_BASE_URL, api_key=openrouter_api_key)\n",
    "ollama_client = AsyncOpenAI(base_url=OLLAMA_BASE_URL, api_key=\"ollama\")\n",
    "\n",
    "# Create a model object to provide when creating an Agent\n",
    "gemini_model = OpenAIChatCompletionsModel(model=\"gemini-2.5-flash\", openai_client=gemini_client)\n",
    "grok_3_model = OpenAIChatCompletionsModel(model=\"grok-3-mini-beta\", openai_client=openrouter_client)\n",
    "llama3_3_model = OpenAIChatCompletionsModel(model=\"llama-3.3-70b-versatile\", openai_client=groq_client)\n",
    "grok_3_via_openrouter_model = OpenAIChatCompletionsModel(model=\"x-ai/grok-3-mini-beta\", openai_client=openrouter_client)\n",
    "llama_3_2_local_model = OpenAIChatCompletionsModel(model=\"llama3.2\", openai_client=ollama_client)\n",
    "```\n",
    "\n",
    "### To use Azure with OpenAI Agents SDK\n",
    "\n",
    "See instructions here:  \n",
    "https://techcommunity.microsoft.com/blog/azure-ai-services-blog/use-azure-openai-and-apim-with-the-openai-agents-sdk/4392537\n",
    "\n",
    "Such as this:\n",
    "```python\n",
    "from openai import AsyncAzureOpenAI\n",
    "from agents import set_default_openai_client\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    " \n",
    "# Load environment variables\n",
    "load_dotenv()\n",
    " \n",
    "# Create OpenAI client using Azure OpenAI\n",
    "openai_client = AsyncAzureOpenAI(\n",
    "    api_key=os.getenv(\"AZURE_OPENAI_API_KEY\"),\n",
    "    api_version=os.getenv(\"AZURE_OPENAI_API_VERSION\"),\n",
    "    azure_endpoint=os.getenv(\"AZURE_OPENAI_ENDPOINT\"),\n",
    "    azure_deployment=os.getenv(\"AZURE_OPENAI_DEPLOYMENT\")\n",
    ")\n",
    " \n",
    "# Set the default OpenAI client for the Agents SDK\n",
    "set_default_openai_client(openai_client)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CrewAI setup\n",
    "\n",
    "Here's Crew's docs for LLM connections with the model names to use for all models. As student Sadan S. pointed out (thank you!), it's worth knowing that for Google you need to use the environment variable `GEMINI_API_KEY` instead of `GOOGLE_API_KEY`:\n",
    "\n",
    "https://docs.crewai.com/concepts/llms\n",
    "\n",
    "And here's their tutorial with some more info:\n",
    "\n",
    "https://docs.crewai.com/how-to/llm-connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LangGraph setup\n",
    "\n",
    "To use LangGraph with Ollama (and follow similar for other models):  \n",
    "https://python.langchain.com/docs/integrations/chat/ollama/#installation\n",
    "\n",
    "First add the package:  \n",
    "`uv add langchain-ollama`\n",
    "\n",
    "Then in the lab, make this replacement:   \n",
    "```python\n",
    "from langchain_ollama import ChatOllama\n",
    "# llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
    "llm = ChatOllama(model=\"gemma3:4b\")\n",
    "```\n",
    "\n",
    "And obviously run `!ollama pull gemma3:4b` (or whichever model) beforehand.\n",
    "\n",
    "Many thanks to Miroslav P. for adding this, and to Arvin F. for the question!\n",
    "\n",
    "## LangGraph with other models\n",
    "\n",
    "Just follow the same recipe as above, but use any of the models from here:  \n",
    "https://python.langchain.com/docs/integrations/chat/\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AutoGen with other models\n",
    "\n",
    "Here's another contribution from Miroslav P. (thank you!) for using Ollama + local models with AutoGen, and Miroslav has a great example showing gemma3 performing well.\n",
    "\n",
    "```python\n",
    "# model_client = OpenAIChatCompletionClient(model=\"gpt-4o-mini\")\n",
    " \n",
    "from autogen_ext.models.ollama import OllamaChatCompletionClient\n",
    " \n",
    "model_client = OllamaChatCompletionClient(\n",
    "    model=\"gemma3:4b\",\n",
    "    model_info={\n",
    "        \"vision\": True,\n",
    "        \"function_calling\": False,\n",
    "        \"json_output\": True,\n",
    "        \"family\": \"unknown\",\n",
    "    },\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Worth keeping in mind\n",
    "\n",
    "1. If you wish to use Ollama to run models locally, you may find that smaller models struggle with the more advanced projects. You'll need to experiment with different model sizes and capabilities, and plenty of patience may be needed to find something that works well. I expect several of our projects are too challenging for llama3.2. As an alternative, consider the free models on openrouter.ai, or the very cheap models that are almost free - like DeepSeek.\n",
    "\n",
    "2. Chat models often do better than Reasoning models because Reasoning models can \"over-think\" some assignments. It's important to experiment. Bigger isn't always better...\n",
    "\n",
    "3. It's confusing, but there are 2 different providers that sound similar!  \n",
    "- Grok is the LLM from Elon Musk's X\n",
    "- Groq is a platform for fast inference of open source models\n",
    "\n",
    "A student pointed out to me that \"Groq\" came first!\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
